blog: why cardinality problems show up too late#2915
Conversation
Draft blog post based on recent discussion around high-cardinality patterns in instrumentation. Focuses on where issues are introduced vs where they become visible in the pipeline. Happy to expand on any specific examples from that discussion if useful during review. Signed-off-by: yupme-bot <yupme112@gmail.com>
nwanduka
left a comment
nwanduka
left a comment
There was a problem hiding this comment.
Thanks for working on this, @yupme-bot 👌. I’ve reviewed it for grammar and clarity, and it looks good to me.
@bboreham, when you get the chance, could you please review it for technical accuracy?
|
|
||
| This creates a gap in the pipeline: | ||
|
|
||
| ``` |
There was a problem hiding this comment.
I wonder how this section will be rendered on the blog.
bboreham
left a comment
bboreham
left a comment
There was a problem hiding this comment.
Overall I think this is reasonable, but I made a number of comments while reading.
Mostly I think you should explain more: write to an audience that isn't already immersed in the detail.
| date: 2026-03-23 | ||
| --- | ||
|
|
||
| High-cardinality metrics are a well-known problem in Prometheus. Most people are familiar with the guidance: avoid labels with unbounded values like user IDs, request IDs, or full URL paths. |
There was a problem hiding this comment.
I think you should define the word "Cardinality" in the blog post, or post a link to somewhere that explains the way in which you are using the word.
For instance it's not this one: https://en.wikipedia.org/wiki/Cardinality_(data_modeling).
|
|
||
| ## The symptom: cardinality shows up late | ||
|
|
||
| In Prometheus, every unique combination of label values creates a new time series. When high-entropy values are used as labels, the number of series can grow quickly. |
There was a problem hiding this comment.
"high-entropy" is another jargon term that deserves a definition.
Earlier you used "unbounded" to mean (I think) the same thing.
|
|
||
| The root cause is usually not in Prometheus itself, but earlier in the pipeline. | ||
|
|
||
| With OpenTelemetry-style instrumentation, it is very easy to attach rich context as attributes: |
There was a problem hiding this comment.
What about this problem is specific to OpenTelemetry?
Did you mean more that it is a general problem, but you are using OpenTelemetry to illustrate?
|
|
||
| Nothing about this looks wrong in isolation. | ||
|
|
||
| But if those values end up as labels downstream, each distinct value becomes a new time series. |
There was a problem hiding this comment.
Why wouldn't they end up as labels? Does that even work?
|
|
||
| But if those values end up as labels downstream, each distinct value becomes a new time series. | ||
|
|
||
| A single line attaching a highly variable value as an attribute can look completely reasonable in a code review. |
There was a problem hiding this comment.
"highly variable" is another synonym for "unbounded" and "high-entropy"?
|
|
||
| ### Guidance exists, but is not visible at the right time | ||
|
|
||
| The docs are clear about avoiding high-cardinality labels, but that guidance is not always present when writing instrumentation. |
There was a problem hiding this comment.
Suggest to link to the specific place in the docs.
|
|
||
| By the time it is noticed, it is often already affecting production systems. | ||
|
|
||
| Most solutions today focus on reducing the impact—normalizing values, limiting label sets, or making attribute-to-label conversion opt-in. |
There was a problem hiding this comment.
"Most solutions today" sets up in my mind an expectation that you will present something different in this post. But I don't think that expectation is met; these seem to be the solutions you present.
| - **Be careful with attribute-to-label conversion** | ||
| Not every attribute needs to become a metric label. |
There was a problem hiding this comment.
I'm not sure what this means. Can you include a link to where readers could find out more?
| - **Treat cardinality as a design concern** | ||
| It is much easier to avoid these issues up front than to fix them later. |
There was a problem hiding this comment.
Can you give, or point to, some specific ways in which to go about this?
E.g. multiply together the cardinality of each independent label.
3 participants
Draft blog post based on recent discussion around high-cardinality patterns in instrumentation.
Focuses on where issues are introduced vs where they become visible in the pipeline.
Happy to expand on any specific examples from that discussion if useful during review.